System for remote viewing and display of a slide

ABSTRACT

A method for creating a virtual slide is provided. A virtual slide is a digital representation of an area of interest of a microscope slide. One method is to use a motorized microscope that can move a specimen with respect to a microscope objective. With such a system, one can capture one or more images through a microscope objective, such that a region of interest is imaged. Each image is then joined together to form a composite or “virtual image.” In one embodiment, after a virtual slide is created, a user may fully utilize the full capabilities of the remote microscope. Among these capabilities is a set of “optical objectives” and “virtual objectives.” Optical objectives are images created by digitizing an image through a microscope objective in real time. Virtual objectives are digitally created magnifications created by utilizing the existing virtual slide data to digitally create a field of view.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a divisional application filed from application Ser. No.10/620,016, filed Jul. 14, 2003, which claims benefit of U.S.Provisional Application No. 60/087,523, filed Jun. 1, 1998, U.S.application Ser. No. 09/323,371, filed Jun. 1, 1999, and U.S.application Ser. No. 10/488,913, filed May 30, 2003. U.S. patentapplication Ser. No. 10/620,016 is incorporated herein by reference.

BACKGROUND OF THE INVENTION

This invention relates to a method and apparatus for viewing remotemicroscope images.

Currently there is increasing demand for pathologist review of samplesat remote locations. There exist multiple systems to address this need.They typically fall within one of two categories: live remote microscopyand virtual slide imaging.

In live remote control microscopy, a user receives images that are takenfrom a slide on a microscope. In virtual slide imaging, a user receivesimages previously captured. Virtual slide systems take one or moreimages of an area of interest and assemble them together (if there ismore than one image) to form a virtual slide. Each of these techniqueshas its advantages. Live remote imaging provides users with the closestapproximation to manual manipulation. Virtual slides allow faster imageviewing, since images are already captured.

Virtual slide systems take one or more images and assemble them to forma “virtual slide.”

However, users in the past were limited in their ability to integratethese technologies. One could only view and manipulate live and virtualimages independently of one another. A user would have to clumsily goback and forth between these two modes of operation to separately lookat the virtual slides and live microscope slides. We present a newmethod that integrates these ideas into one seamless operatingenvironment.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method and apparatus areprovided for the analysis of remote slides in a hybrid live and virtualmedium. Users obtain benefits of each technique in a unifiedenvironment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A virtual slide is a digital representation of an area of interest of amicroscopic slide. A virtual slide can be created multiple ways.

One method is to use a motorized microscope that can move a specimenwith respect to a microscope objective (e.g., a microscope with amotorized stage). With such system, one can capture one or more imagesthrough a microscope objective, such that a region of interest (all orpart of the microscopic slide) is imaged. Each image is then joinedtogether to form a composite or “virtual image.” Multiple methods ofjoining images together are known in the art. One example is when imagesare simply abutted one next to another. However, this method does notgenerally produce virtual slides without seams, because errors such ascamera rotation relative to the axis of motion are difficult to correct.Even with submicron accuracy stages it is, in practice, difficult toobtain consistent positioning. Another method is to utilize overlapbetween adjacent images to edge align images for maximum seamlessness.This can be done by sequentially shifting overlapping regions in the xand/or y axis, for example, by a stepping motor, and calculating acorrelation value (or measure of goodness of overlap). The shift whichresults in the best correlation value is then used to join the imagestogether (FIG. 1). While this method can be computationally expensive,it reduces reliance on difficult-to-attain mechanical positioningrequirements and ultimately produces the best images in the sense ofseamlessness.

In another method, the virtual slide is made simply by utilizing animaging device with optics suitable to take a an image of the area ofinterest on the slide in one snapshot. This method is embodied in theform of a conventional digital or analog camera with a macro lens.

This virtual slide can then be used to create a thumbnail view of theslide. To create the thumbnail view, the virtual slide is shrunk inresolution from its original, base resolution to a target resolution. Ifthe target resolution is the same as the base resolution, then the imageis unchanged. However, typically the resolution of the thumbnail desiredis several times smaller than the base resolution.

With the virtual slide created, the user may fully utilize the fullcapabilities of the remote microscope. The user is presented with animage window and a set of control features (FIG. 2). Among these controlfeatures is a set of “optical objectives” and “virtual objectives.”

Optical objectives are images that are created by a camera digitizing animage through a microscope objective (e.g., 10x, 20x, or 40x) in realtime (i.e., an image is captured at the time the user requests theimage). Virtual objectives are digitally created magnifications creatednot by digitizing in real time, but rather by utilizing the existingvirtual slide data to digitally create a field of view.

When a user selects one of the optical objectives, a “change objective”command is sent to the microscope. This change objective command canalso, specify additional qualifying information, such as microscope x,y, z positions, exposure setting, compression type and level, and imagedimensions. If additional qualifying information is not sent, then theimplicit qualifying information is the current state of the microscopeor the last specified state. When the microscope receives the command,actions are taken to change the objective lens and to change the stateof the microscope commensurate with the command (e.g., change therelative position of the objective lens relative to the microscopicslide, change exposure, etc.). An image is then digitized, compressed ifso specified, and then transmitted to the user for display.

When a user selects a virtual objective, a virtual objective command issent to the microscope. Similar to an optical objective command, thisvirtual objective command can also specify additional qualifyinginformation, such as microscope x, y, and/or z position. If additionalqualifying information is not sent, then the implicit qualifyinginformation is current microscope state or the last specified state. Aregion of interest is defined by the virtual request—it is the area onthe microscope slide included in the field specified by the coordinatesof the stage x, y and magnification of the command. This region ofinterest may optionally be trimmed such that image information alreadyresiding at the requesting user's view is not retransmitted to the user.

An image of the region of interest can be created from the virtual slidein multiple ways. If the virtual slide is not compressed, the retrievalof image information corresponding to the region of interest can be doneby simply copying data from the virtual slide. If the virtual slide iscompressed, a region corresponding to at least the region of interestcan be decompressed to a raw bitmap from the main compressed image. Ifthe virtual slide was stored as multiple compressed images rather thanone large image, additional efficiencies are possible. For example, onlythose images that contain the desired region of interest need beaccessed for decompression, rather than the entire area of the virtualslide. This enhances performance.

The decompression itself can also be performed in various ways.Strategies such as scaled decoding, as in the case of jpeg typecompression, can be employed to improve the speed of the decompressionby coupling a resolution reduction process with the decompression tospeed up decompression when resolution reduction is required. Once theregion of interest is decompressed, it can then be recompressed using avariety of strategies known to those skilled in the imaging field, whichneed not be the same as the method by which the virtual slide wascompressed.

An alternative type of decompression/recompression step can also be usedinvolving partial decompression. Partial decompression, such as decodingof Huffman-encoded data, as in the case of jpeg, can be performed toproduce raw coefficients, rather than full decompression, which producesa raw bitmap. The raw coefficients corresponding to the area of theregion of interest can then be selected. These coefficients whichcorrespond to the region of interest are then re-encoded. In the case ofjpeg compression, this would involve re Huffman encoding of thecoefficients, rather than in normal full compression, where a dct mustbe performed followed by quantization and then Huffman coding.

Whichever the technique, the result is a compressed region of interest.The compressed region of interest is then transmitted to the user forviewing. The described method is more advantageous than sending theentire virtual slide, as one efficiently sends only that informationrequired by the user.

However, direct transfer of the compressed image without decompressionis feasible when the virtual slide is stored as multiple compressedimages. The compressed images that include the area specified by theregion of interest can be directly transferred to the user, rather thangoing through a decompression/recompression step. The disadvantage isthat one may transfer more information than is needed if, for example,the compressed images are at a higher resolution that the requestedresolution. This can be partially solved by creation of multipleresolution versions of the virtual slide. There are also compressionstrategies available that allow only portions of the compressed imagesto be sent, such that a given resolution can be attained depending onwhich portions of the compressed image one chooses to send (e.g.,progressive encoding). However, there is still the issue that the regionof interest only partially covers the area of the compressed image. Inthis case, direct transfer of the image results in inefficiently sendingdata including both the region of interest and data outside the regionof interest to the user.

With this invention, users are afforded a streamlined method ofutilizing the features of virtual and live microscopy techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a drawing showing the overlap between adjacent images duringoptimization of overlap.

FIG. 2 is a photograph of a user's view of the remote microscope,showing a thumbnail view, microscope imaging window, and a set ofmicroscope controls.

While the apparatus and methods of the present invention have beenillustrated in terms of certain embodiments, the invention claimedherein is not limited to embodiments disclosed in this application.Rather, the scope of the invention is defined by the claims attachedhereto.

While the invention has been illustrated and explained herein in termsof certain embodiments the invention is not limited to the specificembodiments disclosed. Rather, the invention is defined by the scope ofthe claims appended hereto.

A method for simultaneously viewing remote microscope images comprisingvirtual and live images and producing a seamless view of live andvirtual images comprises: providing a virtual slide; providing a livemicroscope slide; and automatically and sequentially shiftingoverlapping regions of images from the slides to obtain the optimalimage is disclosed.

1. A system for remote viewing of a slide, comprising: a microscopysystem comprising a microscope having a stage to receive the slide and acamera to couple with the microscope; a single display to show an imageof a portion of the slide and to show a plurality of control features,the plurality of control features comprising: one or more firstselections to command the microscopy system to capture a real timeversion of an other image of at least part of the slide and to transmitthe captured image to the single display; and one or more secondselections to command a previously-captured version of the other imageto be transmitted to the single display.
 2. The system of claim 1,wherein the single display is further to show a thumbnail view of theslide.
 3. The system of claim 2, wherein the thumbnail view of the slidecomprises multiple images joined together.
 4. The system of claim 1,wherein the one or more first selections are further to specify amicroscope objective of the microscope through which the microscopesystem is to capture the real time version of the other image.
 5. Thesystem of claim 1, wherein the one or more second selections are furtherto specify a magnification of the previously-captured version of theother image to be transmitted.
 6. The system of claim 1, wherein the oneor more second selections are further to specify a region of interest ofthe slide, the region of interest to be shown by the previously-capturedversion of the other image.
 7. The system of claim 6, wherein if theimage of the portion of the slide to be shown on the display comprisespart of the specified region of interest, then the previously-capturedversion of the other image to be transmitted to the single display is tonot include the part of the specified region of interest.
 8. The systemof claim 1, wherein the plurality of control features further compriseone or more third selections to specify qualifying information for atleast one of the real time version of the other image and thepreviously- captured version of the other image.
 9. The system of claim8, wherein the qualifying information comprises an x, y microscopeposition.
 10. The system of claim 8, wherein the qualifying informationcomprises an x, y, z microscope position.
 11. The system of claim 8,wherein the qualifying information comprises image dimensions.
 12. Thesystem of claim 8, wherein the qualifying information comprises anexposure setting.
 13. The system of claim 8, wherein the qualifyinginformation comprises a compression level.
 14. The system of claim 8,wherein the qualifying information comprises a compression type.
 15. Amethod for remote displaying of a slide, comprising: providing a singledisplay showing an image of a portion of the slide and showing aplurality of control features; receiving, by receipt of a selection ofone or more of the control features, a command to obtain an other imageof at least part of the slide, the command specifying the other image aseither previously-captured or for capture; and in response to thecommand: if the command specified the other image aspreviously-captured, retrieving a previously-captured of the other imageand transmitting the previously-captured version to the single display;and if the command specified the other image as for capture, capturingthe other image and transmitting the captured other image to the singledisplay.
 16. The method of claim 15, wherein the single display furthershows a thumbnail view of the slide.
 17. The method of claim 15, whereinthe command specifying the other image as for capture further specifiesa microscope objective, and wherein if the command specified the otherimage as for capture, the other image is captured through the microscopeobjective.
 18. The method of claim 15, wherein the command specifyingthe other image as previously-captured further specifies a magnificationof the other image, and wherein if the command specifies the other imageas previously- captured, the previously-captured other image that isretrieved is at the specified magnification.
 19. The method of claim 15,wherein the command further specifies the other image with qualifyinginformation.
 20. The method of claim 19, wherein the qualifyinginformation comprises an x, y microscope position.
 21. The method ofclaim 19, wherein the qualifying information comprises an x, y, zmicroscope position.
 22. The method of claim 19, wherein the qualifyinginformation comprises image dimensions.
 23. The method of claim 19,wherein the qualifying information comprises an exposure setting. 24.The method of claim 19, wherein the qualifying information comprises acompression level.
 25. The method of claim 19, wherein the qualifyinginformation comprises a compression type.
 26. A method for remoteviewing of a slide, comprising: receiving an image of a portion of theslide and showing a plurality of control features and the image on asingle display; selecting certain of the image control features andthereby requesting an other image specified as eitherpreviously-captured or for capture, the other image of at least part ofthe slide; and receiving the specified other image and showing the otherimage on the single display.
 27. The method of claim 26, wherein thesingle display further shows a thumbnail view of the slide.
 28. Themethod of claim 26, wherein the request further specifies the otherimage with qualifying information.
 29. The method of claim 28, whereinthe qualifying information comprises an x, y microscope position. 30.The method of claim 28, wherein the qualifying information comprises anx, y, z microscope position.
 31. The method of claim 28, wherein thequalifying information comprises image dimensions.
 32. The method ofclaim 28, wherein the qualifying information comprises an exposuresetting.
 33. The method of claim 28, wherein the qualifying informationcomprises a compression level.
 34. The method of claim 28, wherein thequalifying information comprises a compression type.